6S: A Collaborative Web Search Network

6S is a collaborative peer network application, aimed to extend the current model of centralized search engines with large numbers of autonomous, distributed, micro-search engines. Each peer within the 6S network crawls the Web in a focused way, guided by its user’s information context. This way better contextual coverage can be achieved. Each peer also acts within the network by submitting, forwarding, and responding to queries to/from its neighbors. Peers depend on a local adaptive routing algorithm to dynamically change the topology of the peer network and search for the best neighbors to answer their queries. If realized, the 6S network will lead to an “intelligent network” where peers learn continually about users and other peers, thus addressing user search needs in a contextual, personalized, and scalable way. Experimental plan The initial phase of this project was funded by a NSF Career grant. This has proven the viability of the 6S idea via large-scale simulation, and has led to the development of a first working prototype. At the current stage, each peer is identical in functionality and differs only in the focus of its collection; all peers employ the same keyword interface to retrieve and rank results, and the same learning algorithm to track other peers and route queries. Now we propose to deploy such application in the real world and allow for the development of peers with different focus, expertise, and semantic capabilities. Our goal is for each peer to be able to easily implement different algorithms for topical Web crawling, ranking of search results, modeling of other peers, and routing queries based on their semantics. This will enable a new level of community search that may lead to seamless integration of diverse and novel search applications. Potential benefits from and to UIMA The proposed network can leverage a much broader community than is possible with today’s search engines, even those employing social collaboration. Anyone can develop a search service for any concept, from topical/vertical search engines, to specialized semantic Web services such as travel agents, to universal search—major search engines like Google and Yahoo could participate, and probably would quickly become hubs. We will explore the use of UIMA to enable this vision, from facilitating the design and development of search services (crawling, semantic retrieval, result ranking) to managing the social search network (learning about other peers, maintaining neighbor profiles, routing queries). We will study how UIMA could be used easily by a community of people building specialized search services, and how the social search network would discover and incorporate the semantic capabilities of new search services that the community comes up with. This may require the development of tools to make UIMA accessible to a broad community of people, not just highly skilled programmers. Finally we will study how to integrate UIMA with text analysis (indexing, retrieval, ranking) algorithms provided by different open source projects, such as Apache Lucene/Nutch, that are employed in the current 6S prototype. 6S: A Collaborative Web Search Network 1